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Purpose  of  this  Presentation 

To  offer  a  potential  set  of  TRL  descriptions  for  use  in 
assessing  Practice-Based  Technologies  (PBTs) 


To  provide  people  who  are  thinking  about  using  TRLs  in 
different  environments  with  ideas  on  how  to  go  about 
analyzing  your  context  to  see  if  TRLs  would  be  an 
applicable  concept 
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What  are  PBTs  (Practice-Based 


Technologies)? 

Practices  > 

Processes 

Methods  L 

Approaches  f 

Frameworks  (for 
the  above)  J 


e.g. 

•Product  Line 
Practices 

•CMMI  (framework) 
•Acquisition  practices 
•Transition  processes 


Versus  non-PBTs: 

Hardware 

Software 

Embedded  systems 

e.g.  Biomedical  devices 
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SEI  View  of  (any)  Technology 
Implementation  Risk 

if)  i  L 


??? 

Low  Risk 

High  Risk 

??? 

Increasing  adopter  readiness 
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Why  Should  You  Care  about 
Applying  TRLs  to  PBTs? 

Improvement  of  acquisition,  engineering,  and  management 
practices  all  require  the  implementation  of  PBTs 

Knowing  the  “readiness”  of  a  PBT  could  potentially  be  helpful  (if 
we  can  come  up  with  a  valid  characterization)  in  managing  its 
implementation  risks: 

•  “early”  technologies  may  be  suitable  for  some  adopters,  but 
require  additional  investment  (to  mature)  for  others 

•  “mature”  technologies  may  be  suitable  for  some,  but  offer  no 
competitive  advantage  to  others  (because  everyone  has 
access  to  it) 
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DoD  Technology  Readiness  Levels 
Reminder 

A  scale  from  1  to  9  used  to  assess  technology  maturity* 

1 .  Basic  principles  observed  and  reported. 

2.  Technology  concept  and/or  application  formulated. 

3.  Analytical  and  experimental  critical  function  and/or 
characteristic  proof  of  concept. 

4.  Component  and/or  breadboard  validation  in  laboratory 
environment. 

5.  Component  and/or  breadboard  validation  in  relevant 
environment. 

6.  System/subsystem  model  or  prototype  demonstration  in  a 
relevant  environment. 

7.  System  prototype  demonstration  in  an  operational 
environment. 

8.  Actual  system  completed  and  qualified  through  test  and 
demonstration. 

9.  Actual  system  proven  through  successful  mission  operations. 


*DoD  Interim  Defense  Acquisition  Guidebook,  October  30,  2002 
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Why  New  TRL  Descriptions 
Specifically  for  PBTs? 

TRL  users  find  current  description  difficult  to  interpret  for  non¬ 
hardware/system  technologies 

•  e.g.  software,  medical,  practices 

•  Study  by  SEI  and  Army  CECOM  in  2002  showed  TRLs  also 
not  readily  applied  to  information  assurance  PBTs 

TRLs  have  gone  a  good  ways  beyond  the  “general”  TRLs 
originally  expressed: 

•  Army  developed  TRL  descriptions  for  software 

•  Army  Medical  Research  and  Materiel  Command  developing 
TRL  descriptions  for  biomedical  technologies 

•  AFRL  (Bill  Nolte)  is  maturing  a  software  tool  for  assessing 
TRLs  along  multiple  dimensions 
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TRLs  Only  Address  One  Side... 

Especially  with  PBTs,  TRLs  are  only  one  side  of  the  equation: 

•  Technology  maturity  is  worthless  without  adopter  readiness 

•  The  “fit”  of  the  technology  characteristics  that  affect  adopter 
readiness  is  at  least  as  important  as  any  inherent  maturity  of 
the  technology 

-  SEI  has  developed  a  Readiness  and  Fit  Analysis  (RFA) 
technique  for  helping  organizations  understand  adoption 
risks  based  on  the  fit  of  their  organizational  characteristics 
with  the  assumptions  inherent  in  a  particular  technology 

-  We  see  this  is  as  a  more  productive  direction  for  our 
research  related  to  technology  implementation  than  TRL 
assessment  per  se 
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Our  Approach 


Each  TRL  consists  of 
•  a  Definition,  meant  to  be 
technology-independent 


•  a  more  detailed,  technology- 
dependent  Description 


1.  Basic  principles 
observed  and 
reported 


Lowest  level  of  technology  readiness. 
Scientific  research  begins  to  be  translated 
into  applied  research  and  development. 
Examples  might  include  paper  studies  of  a 
technology’s  basic  properties _ 


Our  approach  was  to  modify  the  Description  for 
each  level,  leaving  the  Definition  as  is. 
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Caveats 

The  Definitions  are  not  really  technology-independent 
(e.g.,  the  term  “breadboard”)  but  for  those  who  want  to  use 
TRLs  to  assess  non-hardware/system  technologies,  they’ll 
have  to  live  with  it  if  they  want  to  be  compliant  with  the 
TRL  scale 

TRLs  are  not  the  only  criteria  that  support  technology 
management,  they  are  just  one  of  numerous  criteria 
•  Users  in  the  SEI/CECOM  study  estimated  the  TRL 
scale  provides  them,  at  most,  30%  of  their  decision 
criteria 

This  begs  a  question:  should  TRLs  be  expanded  to 
appropriately  become  more  of  the  decision  criteria,  or 
should  TRL  users  be  consistently  explicit  about  the 
expectation  of  what  other  decision  criteria  should  be 
involved  in  different  decision  contexts? 
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The  2  Dimensions  Addressed  by  DoD 
TRLs 

For  hardware/systems,  TRLs  1-9  depict  the  following 
general  progression  in  readiness: 

•  The  environment  in  which  the  technology  can  function 
becomes  more  representative  of  the  final  operational 
environment 

-  from  paper  studies  through  laboratory  setup, 
simulated  environments,  to  mission  operations 

•  The  completeness  of  the  technology  increases 

-  from  basic  properties  through  breadboard 
components,  integrated  components,  prototype,  to 
final  form 
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What  Does  this  Mean  for  PBTs? 

The  environment  in  which  the  technology  can  function 
becomes  more  representative  of  the  final  operational 
environment  (a  community  of  users) 

-  for  PBTs  this  means  the  community  of  users 
expands  from  initial  risk  takers  to  more  mainstream 
members  of  the  community 

The  completeness  of  the  technology  increases 

-  For  PBTs  this  means  the  technology  progresses 
from  defined  basic  properties  through  defined  core 
practices ,  implementation  mechanisms,  best 
practices,  to  a  body  of  knowledge 
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Key  Differences 

The  operating  environment  for  PBTs  is 
people/organizations/community,  not 
hardware/systems 

PBT  environment  is  more  mutable, 
malleable,  in  flux 

These  differences,  and  our  experience 
of  the  innate  2  dimensional  nature  of 
technology  adoption,  makes  us 
somewhat  nervous  about  the  long  term 
utility  of  TRLs  for  PBTs 
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PBT  Corollaries  -  SEI  draft 


TRL 

HW/System 

Practice-Based  Technologies 

1 

Scientific  research,  paper 
studies 

Scientific,  behavioral,  and  market  research,  paper 

studies 

2 

Practical,  speculative 
applications  invented 

Practical,  speculative  applications  invented,  potential 
user  communities  identified 

3 

Active  R&D  initiated, 
analytical  and  lab  studies  of 
components 

Active  R&D  initiated,  critical  elements  identified  and 
demonstrated  with  innovative  users 

4 

Basic  components 
integrated,  lab  environment 

Basic  elements  integrated  to  form  core  PBT,  visionary 
leaders  used  to  demonstrate  value  and  transitionability 

5 

Integrated  components 
demonstrated  in  simulated 
environment 

Prototypes  of  implementation  mechanisms 
established,  demonstrated  with  core  PBT  for  pragmatic 
users  in  simulated  environments,  such  as  role-based 

workshops 

6 

Prototype  tested  in  relevant 
environment 

Implementation  mechanisms  refined  and  integrated 
with  core  PBT,  demonstrated  in  relevant  environments, 
e.g.,  pilot  settings 

7 

Actual  system  prototype  in 
operational  environment 

Implementation  needs  of  mainstream  users  identified 
and  integrated  into  the  prototype,  operational  use  by 
relevant  users  demonstrated  across  the  community 

8 

Final  form  proven  to  work  in 
operational  environment 

Technology  picked-up  for  wide-spread  rollout  across 
the  community 

9 

Actual  application  running 
under  mission  conditions 

PBT  use  is  considered  routine  within  community;  best 
practices  and  body  of  knowledge  are  in  place 
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Testing  Our  PBT  Corollaries 
Using  a  Retrospective  Approach 

Before  using  TRLs  for  PBTs  in  a  predictive  manner,  we  believe 
it  prudent  to  apply  them  retrospectively  to  see  if  the  PBT  TRLs 
provide  insights  into  the  evolution  of  a  technology  that  we  have  a 
long  history  with 

•  SW-CMM  was  selected  as  a  PBT  that  has  sufficient  history  to 
investigate  the  insights  that  could  be  gained  with  this 
approach 

Notable  results  of  analysis: 

•  Use  of  the  retrospective  process  helped  us  to  refine  some  of 
the  boundaries  among  the  draft  TRLs 

•  Generally  belief  is  that  we  were  able  to  characterize  relevant 
aspects  of  SW-CMM  evolution 

•  Still  struggling  somewhat  with  how  to  deal  with  technology 
“upgrades”  (ie  SW-CMM  CMMI)  in  PBT  context 
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Example:  SW-CMM  -1 


TRL  # 

Key  Characteristics 

SW-CMM  based  Improvement 
Example 

Nominal 

Timeframe 

1 

Scientific,  behavioral,  and 
market  research,  paper  studies 

IBM  software  framework 
research,  Crosby  research, 
Humphrey  proposal  of  5-level 
maturity  framework 

1985-1987 

2 

Practical,  speculative 
applications  invented,  potential 
user  communities  identified 

Initial  questionnaire 
developed/published  (87-TR-13), 
DoD  and  its  sw-intensive  system 
suppliers  identified 

1986-1987 

3 

Active  R&D  initiated,  critical 

SPA,  87-TR-13  used  with  large 

1987-1989 

elements  identified  and 

DoD  organizations  and 

demonstrated  with  innovative 

contractors;  Managing  the  SI/I/ 

users 

Process  book  published 
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Example:  SW-CMM  -2 


4 

Basic  elements  integrated  to 
form  core  PBT,  visionary 
leaders  used  to  demonstrate 
value  and  transitionability 

SW-CMM  initial  design 
prototyped/tested 

1989-1991 

5 

Prototypes  of  implementation 
mechanisms  established, 
demonstrated  with  core  PBT 
for  pragmatic  users  in 
simulated  environments,  such 
as  role-based  workshops 

SW-CMM  vl.O  published; 
piloted  with  wider  user  base; 
SPA  and  SCE  used  to  feed 
back  info  to  CMM  dev  team; 
SEPG  workshop  becomes 
SEPG  conference 

1991-1993 

6 

Implementation  mechanisms 
refined  and  integrated  with 
core  PBT,  demonstrated  in 
relevant  environments,  e.g., 
pilot  settings 

SW-CMM  vl.1  published;  Intro 
training,  CBA-IPI  and  lead 
appraiser  program  developed; 
ROI  case  studies  published 

1993-1995 
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Example:  SW-CMM  -3 


8 


9 


Implementation  needs  of 
mainstream  users  identified 
and  integrated  into  the 
prototype,  operational  use  by 
relevant  users  demonstrated 
across  the  community 


Technology  picked-up  for 
wide-spread  rollout  across  the 
community 


Transition  Partner,  CBA-IPI,  1993-1997 
SCE  3.0,  Intro  TTT  established; 

SW  measurement  books 
published;  process  support 
(proc  defn,  MPI)  courses 
developed;  SW-CMM  v2.0 
drafted 

“YAMMs”  phenomenon;  high  1995-1997 
maturity  workshops  established; 
principles  for  CMM  established; 

SW-CMM  v2.0  chosen  as  basis 
for  CMM  I  framework 


PBT  use  is  considered  routine 
within  community,  best 
practices  and  body  of 
knowledge  are  in  place,  may 
involve  incorporation  of  the 
technology  into  community 
guidance  and  policy 


Incorporation  of  CMM  concepts  1997-2001 
into  ISO  15504;  over  60  orgns 
invited  to  2001  high  maturity 
workshop;  noticeable 
improvement  in  maturity  profile 
for  intended  community;  SW- 
CMM  subsumed  into  CMMI 
(broadening  overall  community) _ 
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Summary  and  Next  Steps 

Potential  draft  of  TRL  Descriptions  for  PBTs  has  been 
defined  here 

•  No  funding  is  allocated  for  going  beyond  this  stage 

Community  feedback  and  participation  welcome  (send 
email  to  cpg@sei.cmu.edu  or  smq@sei.cmu.edu  ) 

Next  steps  possibilities: 

•  Incorporate  a  PBT  TRL  assessment  as  part  of 
Readiness  &  Fit  Adoption  Risk  Analysis 

•  Further  explore  the  effects  of  using  a  single  scale  to 
represent  a  (at  least!)  two  dimensional  situation 
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For  more  information... 

On  PBT  TRLs: 

•  SuZ  Garcia,  smq@sei.cmu.edu 

•  Caroline  Graettinger,  cpq@sei.cmu.edu 

•  Eileen  Forrester,  ecf@sei.cmu.edu 

On  Readiness  &  Fit  Analysis  (RFA): 

•  SuZ  Garcia,  smq@sei.cmu.edu 
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